library(readxl)
library(tidyverse)
library(leaflet)
library(sf)
library(tigris)Lab 10: APIs
Introduction
In this lab, we will write queries to access subsets of a very large dataset on the NYC Open Data Portal. We will practice all of the standards we have learned in the course so far in visualizing and wrangling the resulting data.
Learning Goals
- Write API Queries
- Recognize different HTTP Response Codes
Review of Key Terms
Tip
You may wish to reference this reference guide when completing this lab.
- Application Programming Interface (API)
-
an mechanism that allows programmers or other systems (or users) to access resources from or post data to an online data service
- HTTP
-
a Web protocol for transmitting documents (e.g. Web pages, images, datasets, etc.) between web servers and browsers
- Endpoint
-
An exposed portion of an online data service that clients can assess via API queries
NYC 311 Service Requests
311 is a dialing code set aside in many cities across the US to field calls about requests for municipal service – fixing potholes, reporting noise complaints, reporting parking infractions, etc. NYC has one of the most comprehensive 311 programs in the country, fielding calls to be routed to one of the city’s 77 agencies. Unlike in many other cities, New Yorkers can call 311 to report sexual harassment in taxi cabs or to report tenant rights issues (like a landlord failing to address a lack of heat or hot water). In October 2011, data documenting each request for service New Yorkers had made to 311 began to be published daily to the city’s public-facing open data portal. Since then, the dataset has become one of the largest and most complex archived and maintained on the portal, which includes datasets covering topics from restaurant health inspections, to crime statistics, to construction permits. Each row in the 311 dataset details one anonymized and geocoded request - the type of request that was made, when it was made, the location where the incident occurred, and how the relevant NYC agency responded. As of November 2022, the dataset has over 31 million rows, representing service requests made to 311 since 2010.
When the 311 program was introduced, it was celebrated for its bottom-up approach to producing empirical evidence about problems facing New Yorkers; the program buttressed a persuasive techno-liberal imaginary, suggesting that fair and unbiased representation of quality of life concerns would emanate through the crowd-sourced data. When wrangled and visualized, the data can produce persuasive narratives about the state of equity and quality of life in communities across NYC, and thus a number of communities now leverage the data to legitimate claims. Community boards regularly cite 311 statistics about noise, infrastructure, and rodent issues when preparing district budget requests. Activists present the data to city legislators when advocating for or against new housing and transportation laws. Cyclists have produced apps that track complaints about the blocked bicycle lanes in New York. Journalists regularly reference the dataset to report which communities have received the most noise complaints, have had the most restaurants not following Covid-19 orders, or have had the most rat sightings. In 2018, eight pieces of legislation introduced by City Council required data about various issues – from rodent complaints, to noise complaints, to complaints about sexual harassment in taxis – be reported to the city agencies responsible for addressing those issues.
Yet, while representatives acknowledge 311 data to be a useful form of evidence for quality of life issues, they can be hesitant to rely on 311 statistics alone to measure urban problems. Communities that regularly leverage 311 open data warn that the data does not “represent” the saturation of problems in the city but that they represent “where people are complaining in the city.”
Mapping the data shows that, for almost all reported issues, 311 complaints are disproportionately high in gentrifying communities. The data under-represents problems in areas where individuals do not know to report, do not have the capacity to report, or fear harassment (from their landlords or bosses) for reporting. Complaints about NYC Housing Authority (NYCHA) are entirely excluded from the data. The data also over-represents problems when weaponized against minority communities and small businesses to consistently report minor and sometimes false noise issues or legal infractions. While this highlights how the dataset is being used to propagate inequity across the city, activists have also recognized that they can leverage 311 in liberatory ways. I’ve interviewed tenant rights groups that will run 311 calling campaigns in housing complexes, which can significantly increase the number of complaints made at a particular address - not necessarily because it is experiencing more issues than the building next to it, but because there has been advocacy around reporting the issues at this location.
Setting Up Your Environment
Install the
httrpackage.Run the code below to load the packages for today’s lab
food_atlas_url <-"https://ers.usda.gov/sites/default/files/_laserfiche/DataFiles/80591/FoodAccessResearchAtlasData2019.xlsx?v=95432"
temp <- tempfile()
download.file(food_atlas_url, temp)
food_atlas <- read_excel(temp, sheet=3)
nm_tracts <- tracts(state = "NM", year = 2019)
|
| | 0%
|
| | 1%
|
|= | 1%
|
|= | 2%
|
|== | 2%
|
|== | 3%
|
|=== | 4%
|
|=== | 5%
|
|==== | 5%
|
|==== | 6%
|
|===== | 7%
|
|===== | 8%
|
|====== | 8%
|
|====== | 9%
|
|======= | 9%
|
|======= | 10%
|
|======= | 11%
|
|======== | 11%
|
|======== | 12%
|
|========= | 13%
|
|========== | 14%
|
|========== | 15%
|
|=========== | 15%
|
|=========== | 16%
|
|============ | 16%
|
|============ | 17%
|
|============= | 18%
|
|============= | 19%
|
|============== | 20%
|
|=============== | 21%
|
|=============== | 22%
|
|================ | 22%
|
|================ | 23%
|
|================ | 24%
|
|================= | 24%
|
|================= | 25%
|
|================== | 25%
|
|================== | 26%
|
|=================== | 27%
|
|=================== | 28%
|
|==================== | 28%
|
|==================== | 29%
|
|===================== | 29%
|
|===================== | 30%
|
|===================== | 31%
|
|====================== | 31%
|
|====================== | 32%
|
|======================= | 32%
|
|======================= | 33%
|
|======================= | 34%
|
|======================== | 34%
|
|======================== | 35%
|
|========================= | 35%
|
|========================= | 36%
|
|========================== | 37%
|
|========================== | 38%
|
|=========================== | 38%
|
|=========================== | 39%
|
|============================ | 39%
|
|============================ | 40%
|
|============================ | 41%
|
|============================= | 41%
|
|============================= | 42%
|
|============================== | 42%
|
|============================== | 43%
|
|=============================== | 44%
|
|=============================== | 45%
|
|================================ | 45%
|
|================================ | 46%
|
|================================= | 47%
|
|================================= | 48%
|
|================================== | 48%
|
|================================== | 49%
|
|=================================== | 49%
|
|=================================== | 50%
|
|=================================== | 51%
|
|==================================== | 51%
|
|==================================== | 52%
|
|===================================== | 52%
|
|===================================== | 53%
|
|====================================== | 54%
|
|====================================== | 55%
|
|======================================= | 56%
|
|======================================== | 57%
|
|======================================== | 58%
|
|========================================= | 58%
|
|========================================= | 59%
|
|========================================== | 59%
|
|========================================== | 60%
|
|=========================================== | 61%
|
|=========================================== | 62%
|
|============================================ | 62%
|
|============================================ | 63%
|
|============================================ | 64%
|
|============================================= | 64%
|
|============================================= | 65%
|
|============================================== | 65%
|
|============================================== | 66%
|
|=============================================== | 67%
|
|=============================================== | 68%
|
|================================================ | 68%
|
|================================================ | 69%
|
|================================================= | 69%
|
|================================================= | 70%
|
|================================================= | 71%
|
|================================================== | 71%
|
|================================================== | 72%
|
|=================================================== | 72%
|
|=================================================== | 73%
|
|==================================================== | 74%
|
|==================================================== | 75%
|
|===================================================== | 75%
|
|===================================================== | 76%
|
|====================================================== | 77%
|
|====================================================== | 78%
|
|======================================================= | 78%
|
|======================================================= | 79%
|
|======================================================== | 80%
|
|======================================================== | 81%
|
|========================================================= | 81%
|
|========================================================= | 82%
|
|========================================================== | 82%
|
|========================================================== | 83%
|
|=========================================================== | 84%
|
|=========================================================== | 85%
|
|============================================================ | 85%
|
|============================================================ | 86%
|
|============================================================= | 87%
|
|============================================================= | 88%
|
|============================================================== | 88%
|
|============================================================== | 89%
|
|=============================================================== | 89%
|
|=============================================================== | 90%
|
|=============================================================== | 91%
|
|================================================================ | 91%
|
|================================================================ | 92%
|
|================================================================= | 93%
|
|================================================================== | 94%
|
|================================================================== | 95%
|
|=================================================================== | 95%
|
|=================================================================== | 96%
|
|==================================================================== | 96%
|
|==================================================================== | 97%
|
|==================================================================== | 98%
|
|===================================================================== | 98%
|
|===================================================================== | 99%
|
|======================================================================| 100%food_atlas_nm <- nm_tracts |> left_join(food_atlas, by = c("GEOID" = "CensusTract"))
food_atlas_nm |> st_crs()Coordinate Reference System:
User input: NAD83
wkt:
GEOGCRS["NAD83",
DATUM["North American Datum 1983",
ELLIPSOID["GRS 1980",6378137,298.257222101,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["latitude",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["longitude",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4269]]#leaflet is 4326
nm_map <- leaflet(width = "100%") %>%
setView(lat = 34, lng = -105.0324, zoom = 6) %>%
addProviderTiles(providers$CartoDB.Positron)
create_chloro_map_num <- function(df, var, breaks) {
pal_num <- colorNumeric(palette="YlOrRd",
domain = df$var)
nm_map |>
addPolygons(data = df,
weight = 0.5,
color = "black",
fillColor = ~pal_bin(var),
fillOpacity = 0.5)
}
create_chloro_map_bin <- function(df, var, breaks) {
pal_bin <- colorBin(palette="YlOrRd",
domain = df$var,
bins = breaks)
nm_map |>
addPolygons(data = df,
weight = 0.5,
color = "black",
fillColor = ~pal_bin(var),
fillOpacity = 0.5)
}
create_chloro_map_quant <- function(df, vari, num) {
pal_quant <- colorQuantile(palette="YlOrRd",
domain = df[[vari]],
n = num)
nm_map |>
addPolygons(data = df,
weight = 0.5,
color = "black",
fillColor = ~pal_quant(df[[vari]]),
fillOpacity = 0.5)
}
create_chloro_map_quant(food_atlas_nm, "PovertyRate", 4)